The first technique for automating the deletion of duplicate records is actually going to mark duplicates for future deletion. I like to have a human eye take a look at the results since you can't undo a record delete. I'd rather have a person make the decision to delete a marked record just as a safety net. So the basic idea here is we're going to loop through all the records and compare each one. In order to do that we're going to have to sort the records, so we'll do it manually first. We'll come down, go to the Table, find the dup check field and then Sort it. And you'll see I've added a few duplicates here but the basic idea is you want to compare Record 1 to Record 2 and see if it's a duplicate; Record 2 to Record 3, check it, 3 to 4 and keep going down the list and mark which ones are duplicates by putting a value into a Duplicate field. Now before we add the Duplicate field to Manage, Database, realize that when we put a value into a field that causes record locking so if anybody else is on the system that record won't get marked as a duplicate so you really need to run this when nobody's on the system; either that, don't worry that maybe some of the records won't get marked as duplicates, it won't be that big of a deal, but if you're concerned you really need to run it after hours when nobody's on the system. So we'll go into Manage, Database, we'll add a field called duplicate. We'll create that as a Text field, click OK and we'll go into Layout Mode and temporarily add it right here so you can see the results. And we'll go back into Browse Mode and now what we're going to do is actually work with the script so let's go in there and recreate what we're talking about, so we'll make a New Script, call it Mark Duplicates. We'll start off with our Perform Script, Allow User Abort, Set Error Capture. And then the first step we're going to do is Freeze Window. That'll make the script run faster. Any loop that's going through records you can prevent that redraw with the Freeze Window and make it significantly faster. So the next thing we're going to do is Sort them. We need to Sort those records, we'll perform without dialog, dup check is already in there so we're set and then we're going to go to the first record because when you Sort you stay on the Current Record so we need to make sure that we go to the first record so we don't miss any and I'd actually add Sort Records again, we need to do Go to First Record, there we go, First, then we can begin our Loop. So I'll throw the Loop in there and inside our Loop we're going to first Set a Variable, that'll be dollar sign Dup and the calculation will be simply a reference to dup check. So, we're going to set that on one record, then we're going to go to the next record, Next and Exit after last and then we can check since we're now on the, where it's just starting we'd be on the first record, set the value here, we go to the second record and then we can test to see if dollar sign Dup equals dup check. If that's true we're going to set a field, that'll be our duplicate field and you can put just about anything in here but I'm going to make it real simple and just put an x in there. There we go. There's your entire script. So, in other words, it's going to go to the first record which, if you're wondering at this point, the first record can never be a duplicate. It's impossible just based on - when you see the results you'll understand why but, we're marking the second and third, let's say we had three duplicates right in a row. The first one wouldn't be a duplicate, we're deciding to keep that as original, the oldest one by creation and then mark the second and third for deletion, so the first one, based on that whole idea, couldn't be a duplicate, so we go to that first record, put the value in there, go to the next record, the second one, check to see if they're duplicates, maybe mark it maybe not, come back up here, we're on the second record, put that dup check value, go to the third record and go through this whole process over and over again. So let's give it a try here. We'll Save it. It's under the Scripts Menu so we're good. And what you normally do here would be to put a Find by Duplicates, by the exclamation point at the beginning of the script but I want to have it run through a bunch of different records here so I can really show you what's going on. But the idea behind that is, if you search for duplicates in a million-record database it's going to be much faster to go through what it thinks are the duplicates rather than a million records and try to check them that way. For our example here we don't have that many records, we're going to go through all of them and see what happens.
TERMS & CONDITIONS OF USE
BY SUBSCRIBING TO THIS SERVICE, YOU ARE CONSENTING TO BE BOUND BY AND ARE BECOMING A PARTY TO THIS AGREEMENT, THE TERMS AND CONDITIONS OF WHICH SHALL PREVAIL IN GOVERNING YOUR RIGHTS OF USE. BY CLICKING THE "BECOME A MEMBER" BUTTON, THE INDIVIDUAL OR ENTITY LICENSING THE PRODUCT ("YOU") IS CONSENTING TO BE BOUND BY AND IS BECOMING A PARTY TO THIS AGREEMENT. IF LICENSEE DOES NOT AGREE TO ALL OF THE TERMS OF THIS AGREEMENT, THE BUTTON INDICATING "BECOME A MEMBER" MUST NOT BE SELECTED, AND LICENSEE MUST NOT INSTALL OR USE THE SOFTWARE.
"VTC" refers to Virtual Training Company,
"You" refers to the user or subscriber.
"Software" refers to the VTC training content and software.
2. LICENSE: VTC hereby grants to You a worldwide, non-royalty bearing, non-exclusive license to use the Software according to the provisions contained herein and subject to payment of the applicable subscription fees.
3. RESTRICTIONS: You may not do any of the following:
Save the Software to Your hard disk or other storage
medium; permit others to use the Software except as specified by addendum;
modify, reverse engineer, decompile, or disassemble the Software; make
derivative works based on the Software; publish or otherwise disseminate
the Software. VTC, Inc., VTC Online University, and the Virtual Training
Company site is owned and operated by VTC, Inc. as a corporation of
All materials on this site are the property of VTC unless otherwise specified. No material from these pages may be copied, reproduced, republished, downloaded, uploaded, posted, transmitted, or distributed in any way. Modification of the materials or use of the materials for any other purpose is a violation of U.S. copyright law and other proprietary rights. For purposes of this Agreement, the use of any such material on any other web site or networked computer environment is prohibited.
4. FEES: The rights granted under this Agreement
are effective only upon payment of the subscription fees, which are
strictly non-refundable other than as expressly provided herein. The
term "monthly subscription" is defined as any 30 day period.
The term "yearly subscription" is defined as one 365 day
period. A yearly subscription ends on the same numerical date as it
began (example July 28, 2004 to July 28, 2005).
The VTC Online University is access to every VTC training tutorial in our library. You pay a flat fee for access to these titles. You are billed according to your renewal selection below, and can renew monthly, yearly, or in any other increment offered. If you choose to be billed monthly, you will be billed every 30 days for the subscription until you request the subscription be cancelled. Our terms of service state that you must cancel a monthly subscription at least two business days before your renewal date. These two days give us enough time to ensure that you will not be charged again.
5. LIMITED WARRANTY: VTC warrants that the Software, if operated as directed, will substantially achieve the functionality described. VTC does not warrant, however, that Your use of the Software will be uninterrupted or that the operation of the Software will be error-free or secure. In addition, the security mechanisms implemented by the Software have inherent limitations, and You must determine that the Software sufficiently meets Your requirements. VTC also warrants that the media containing the Software, if provided by VTC, is free from defects in material from the date You acquired the Software. VTC's sole liability for any breach of this warranty shall be, in VTC's sole discretion: (i) to replace Your defective media or Software; or (ii) to advise You how to achieve substantially the same functionality with the Software as described; or (iii) if the above remedies are impracticable, to refund the subscription fee You paid for the Software. Only if You inform VTC of Your problem with the Software during the applicable subscription period will VTC be obligated to honor this warranty. VTC will use reasonable commercial efforts to repair, replace, advise, or refund pursuant to the foregoing warranty within thirty (30) days of being so notified. If any modifications are made to the Software by You during the warranty period; if the medium is subjected to accident, abuse, or improper use; or if You violate the terms of this Agreement, then this warranty shall immediately terminate. This warranty shall not apply if the Software is used on or in conjunction with hardware or software other than the unmodified version of hardware and software with which the Software was designed to be used as described.
THIS IS A LIMITED WARRANTY, AND IT IS THE ONLY WARRANTY MADE BY VTC OR ITS SUPPLIERS. VTC MAKES NO OTHER WARRANTIES, EXPRESS OR IMPLIED, INCLUDING BUT NOT LIMITED TO WARRANTIES OF MERCHANTABILITY, FITNESS FOR A PARTICULAR PURPOSE, AND NONINFRINGEMENT OF THIRD PARTIES' RIGHTS. YOU MAY HAVE OTHER STATUTORY RIGHTS. HOWEVER, TO THE FULL EXTENT PERMITTED BY LAW, THE DURATION OF STATUTORILY REQUIRED WARRANTIES, IF ANY, SHALL BE LIMITED TO THE ABOVE LIMITED WARRANTY PERIOD. MOREOVER, IN NO EVENT WILL WARRANTIES PROVIDED BY LAW, IF ANY, APPLY UNLESS THEY ARE REQUIRED TO APPLY BY STATUTE NOTWITHSTANDING THEIR EXCLUSION BY CONTRACT. NO DEALER, AGENT, OR EMPLOYEE OF VTC IS AUTHORIZED TO MAKE ANY MODIFICATIONS, EXTENSIONS, OR ADDITIONS TO THIS LIMITED WARRANTY.
6. PROPRIETARY RIGHTS: VTC reserves all proprietary rights in and to the Software, is protected by copyright and other intellectual property laws and by international treaties. VTC, Inc.
Trademark Notice: VTC, Virtual Training Company,
Inc., The VTC Logo, and VTC Online University, are trademarks of VTC,
Inc. All other company and product names may be trademarks of their
The information contained herein is subject to change without notice. Copyright © 1995 - 2005 VTC, Inc. All rights reserved.
7. TERMINATION: This Agreement shall automatically terminate if You fail to comply with the restrictions described herein. Your obligations to pay outstanding subscription fees shall survive any termination of this Agreement.
8. LIMITATION OF LIABILITY: UNDER NO CIRCUMSTANCES
AND UNDER NO LEGAL THEORY, TORT, CONTRACT, OR OTHERWISE, SHALL VTC
OR ITS SUPPLIERS OR RESELLERS BE LIABLE TO YOU OR ANY OTHER PERSON
FOR ANY INDIRECT, SPECIAL, INCIDENTAL, OR CONSEQUENTIAL DAMAGES OF
ANY CHARACTER, INCLUDING WITHOUT LIMITATION, DAMAGES FOR LOSS OF GOODWILL,
WORK STOPPAGE, COMPUTER FAILURE OR MALFUNCTION, OR ANY AND ALL OTHER
COMMERCIAL DAMAGES OR LOSSES. IN NO EVENT WILL VTC BE LIABLE FOR ANY
DAMAGES IN EXCESS OF THE AMOUNT VTC RECEIVED FROM YOU FOR A LICENSE
TO THE SOFTWARE, EVEN IF VTC SHALL HAVE BEEN INFORMED OF THE POSSIBILITY
DAMAGES, OR FOR ANY CLAIM BY ANY OTHER PARTY. THIS LIMITATION OF LIABILITY SHALL NOT APPLY TO LIABILITY FOR DEATH OR PERSONAL INJURY RESULTING FROM VTC'S NEGLIGENCE TO THE EXTENT APPLICABLE LAW PROHIBITS SUCH LIMITATION. SOME JURISDICTIONS DO NOT ALLOW THE EXCLUSION OR LIMITATION OF INCIDENTAL OR CONSEQUENTIAL DAMAGES, SO THIS EXCLUSION AND LIMITATION MAY NOT APPLY TO YOU.
9. Links To Other Materials: Linked sites found at the VTC site are not under the control of VTC, and we are not responsible for the content of any linked site or any link contained in a linked site. VTC may change links based solely on our discretion, and we reserve the right to terminate any link or linking program at any time. VTC does not, by linking to sites, endorse companies or products to which it links and reserves the right to note as such on its web pages. If you decide to access any of the third party sites linked to this site, you do this entirely at your own risk.
Forums, and Chat are not always screened by VTC, and we are not responsible for the content of any public or open forum content at the site. VTC may change these public forums based solely on our discretion, and we reserve the right to terminate any forum at any time. VTC does not, by allowing these forums, endorse companies or products which may be mentioned in these forums, and reserves the right to note as such on its web pages. If you decide to access any of the public forums in this site, or linked to this site, you do this entirely at your own risk.
9. GOVERNING LAW & DISPUTE RESOLUTION: This Agreement is governed by Virginia law. All disputes between You and VTC shall be finally resolved through arbitration in Winchester, Virginia. This site is controlled by VTC from its offices within the United States of America. VTC makes no representation that materials in the site are appropriate or available for use in other locations, and access to them from territories where their content is illegal is prohibited. Those who choose to access this site from other locations do so on their own initiative and are responsible for compliance with applicable local laws. You may not use or export the Materials in violation of U.S. export laws and regulations. Any claim relating to the Materials shall be governed by the internal substantive laws of the Commonwealth of Virginia, USA.
VTC may revise these Terms at any time by updating this posting. You should visit this page from time to time to review the then-current Terms because they are binding on you. Certain provisions of these Terms may be superseded by expressly designated legal notices or terms located on particular pages at this Site.
If you have any questions regarding this policy,
or your information specifically,
you may email us at:firstname.lastname@example.org.